Migrate follower restart experiment #269
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In order to be able to deploy per default the benchmark process and remove the hurdle between two different usages, we split of the deploy process and multi-version into two sub-commands. Deployment distribution experiments has been adjusted and tested
Migrates the follower restart experiment and tested with integration test
ChrisKujawa
force-pushed
the
zell-migrate-follower-restart
branch
from
6d88079 to
72c2c3b
Compare
This was referenced Dec 6, 2022
Merged
deepthidevaki
approved these changes
Dec 7, 2022
|
Yeah sorry @deepthidevaki I did it similar to #268
|
9 tasks
ChrisKujawa
added a commit
that referenced
this pull request
Dec 7, 2022
depends #269 related to #237 ----- Migrates the follower terminate experiment and tested with integration test Log output: ``` Deploy file bpmn/chaos/actionRunner.bpmn (size: 8788 bytes). Deployed process model bpmn/chaos/actionRunner.bpmn successful with key 2251799813685249. Deploy file bpmn/chaos/chaosExperiment.bpmn (size: 21403 bytes). Deployed process model bpmn/chaos/chaosExperiment.bpmn successful with key 2251799813685251. Deploy file bpmn/chaos/chaosToolkit.bpmn (size: 11031 bytes). Deployed process model bpmn/chaos/chaosToolkit.bpmn successful with key 2251799813685253. Create ChaosToolkit instance Open workers: [zbchaos, readExperiments]. Handle read experiments job [key: 2251799813685265] Read experiments successful, complete job with: {"experiments":[{"contributions":{"availability":"high","reliability":"high"},"description":"Zeebe should be fault-tolerant. Zeebe should be able to handle followers terminations.","method":[{"name":"Terminate follower of partition 1","provider":{"arguments":["terminate","broker","--role","FOLLOWER","--partitionId","1"],"path":"zbchaos","type":"process"},"type":"action"}],"rollbacks":[],"steady-state-hypothesis":{"probes":[{"name":"All pods should be ready","provider":{"arguments":["verify","readiness"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"},{"name":"Can deploy process model","provider":{"arguments":["deploy","process"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"},{"name":"Should be able to create process instances on partition 1","provider":{"arguments":["verify","instance-creation","--partitionId","1"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"}],"title":"Zeebe is alive"},"title":"Zeebe follower restart non-graceful experiment","version":"0.1.0"},{"contributions":{"availability":"high","reliability":"high"},"description":"This fake experiment is just to test the integration with Zeebe and zbchaos workers","method":[{"name":"Show again the version","pauses":{"after":5},"provider":{"arguments":["version"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"action"}],"rollbacks":[],"steady-state-hypothesis":{"probes":[{"name":"Show version","provider":{"arguments":["version"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"}],"title":"Zeebe is alive"},"title":"This is a fake experiment","version":"0.1.0"}]}. Handle zbchaos job [key: 2251799813685328] Running command with args: [verify readiness] Connecting to zell-chaos Running experiment in self-managed environment. All Zeebe nodes are running. Handle zbchaos job [key: 2251799813685374] Running command with args: [deploy process] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Deploy file bpmn/one_task.bpmn (size: 2526 bytes). Deployed process model bpmn/one_task.bpmn successful with key 2251799813685431. Deployed given process model , under key 2251799813685431! Handle zbchaos job [key: 2251799813685418] Running command with args: [verify instance-creation --partitionId 1] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 4503599627372098 on partition 2, required partition 1. Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 6755399441057346 on partition 3, required partition 1. Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 2251799813686915 on partition 1, required partition 1. The steady-state was successfully verified! Handle zbchaos job [key: 2251799813685467] Running command with args: [terminate broker --role FOLLOWER --partitionId 1] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Found Broker zell-chaos-zeebe-2 as FOLLOWER for partition 1. Terminated zell-chaos-zeebe-2 Handle zbchaos job [key: 2251799813685513] Running command with args: [verify readiness] Connecting to zell-chaos Running experiment in self-managed environment. Pod zell-chaos-zeebe-2 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Pending, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. Pod zell-chaos-zeebe-2 is in phase Running, but not ready. Wait for some seconds. All Zeebe nodes are running. Handle zbchaos job [key: 2251799813685852] Running command with args: [deploy process] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Deploy file bpmn/one_task.bpmn (size: 2526 bytes). Deployed process model bpmn/one_task.bpmn successful with key 2251799813685431. Deployed given process model , under key 2251799813685431! Handle zbchaos job [key: 2251799813685896] Running command with args: [verify instance-creation --partitionId 1] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 4503599627372125 on partition 2, required partition 1. Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 6755399441057377 on partition 3, required partition 1. Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 2251799813686946 on partition 1, required partition 1. The steady-state was successfully verified! Handle zbchaos job [key: 2251799813685990] Running command with args: [version] zbchaos development (commit: HEAD) Handle zbchaos job [key: 2251799813686033] Running command with args: [version] zbchaos development (commit: HEAD) Handle zbchaos job [key: 2251799813686179] Running command with args: [version] zbchaos development (commit: HEAD) Instance 2251799813685255 [definition 2251799813685253 ] completed --- PASS: Test_ShouldBeAbleToRunExperiments (41.89s) ```
ChrisKujawa
added a commit
that referenced
this pull request
Dec 7, 2022
blocked by #269 blocked by #270 related to #237 ------ Migrated the leader restart experiment to the zbchaos toolkit Log output: ``` Deployed process model bpmn/chaos/actionRunner.bpmn successful with key 2251799813685249. Deploy file bpmn/chaos/chaosExperiment.bpmn (size: 21403 bytes). Deployed process model bpmn/chaos/chaosExperiment.bpmn successful with key 2251799813685251. Deploy file bpmn/chaos/chaosToolkit.bpmn (size: 11031 bytes). Deployed process model bpmn/chaos/chaosToolkit.bpmn successful with key 2251799813685253. Create ChaosToolkit instance Open workers: [zbchaos, readExperiments]. Handle read experiments job [key: 2251799813685265] Read experiments successful, complete job with: {"experiments":[{"contributions":{"availability":"high","reliability":"high"},"description":"This fake experiment is just to test the integration with Zeebe and zbchaos workers","method":[{"name":"Show again the version","provider":{"arguments":["version"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"action"}],"rollbacks":[],"steady-state-hypothesis":{"probes":[{"name":"Show version","provider":{"arguments":["version"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"}],"title":"Zeebe is alive"},"title":"This is a fake experiment","version":"0.1.0"},{"contributions":{"availability":"high","reliability":"high"},"description":"Zeebe should be fault-tolerant. Zeebe should recover after a partition leader was restarted gracefully.","method":[{"name":"Restart leader of partition 1","provider":{"arguments":["restart","broker","--role","LEADER","--partitionId","1"],"path":"zbchaos","type":"process"},"type":"action"}],"rollbacks":[],"steady-state-hypothesis":{"probes":[{"name":"All pods should be ready","provider":{"arguments":["verify","readiness"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"},{"name":"Can deploy process model","provider":{"arguments":["deploy","process"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"},{"name":"Should be able to create process instances on partition 1","provider":{"arguments":["verify","instance-creation","--partitionId","1"],"path":"zbchaos","timeout":900,"type":"process"},"tolerance":0,"type":"probe"}],"title":"Zeebe is alive"},"title":"Zeebe Leader restart gracefully experiment","version":"0.1.0"}]}. Handle zbchaos job [key: 2251799813685328] Running command with args: [version] zbchaos development (commit: HEAD) Handle zbchaos job [key: 2251799813685374] Running command with args: [version] zbchaos development (commit: HEAD) Handle zbchaos job [key: 2251799813685417] Running command with args: [version] zbchaos development (commit: HEAD) Handle zbchaos job [key: 2251799813685508] Running command with args: [verify readiness] Connecting to zell-chaos Running experiment in self-managed environment. All Zeebe nodes are running. Handle zbchaos job [key: 2251799813685551] Running command with args: [deploy process] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Deploy file bpmn/one_task.bpmn (size: 2526 bytes). Deployed process model bpmn/one_task.bpmn successful with key 2251799813685431. Deployed given process model , under key 2251799813685431! Handle zbchaos job [key: 2251799813685598] Running command with args: [verify instance-creation --partitionId 1] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 2251799813689038 on partition 1, required partition 1. The steady-state was successfully verified! Handle zbchaos job [key: 2251799813685644] Running command with args: [restart broker --role LEADER --partitionId 1] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Found Broker zell-chaos-zeebe-2 as LEADER for partition 1. Restarted zell-chaos-zeebe-2 Handle zbchaos job [key: 2251799813685690] Running command with args: [verify readiness] Connecting to zell-chaos Running experiment in self-managed environment. All Zeebe nodes are running. Handle zbchaos job [key: 2251799813685732] Running command with args: [deploy process] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Deploy file bpmn/one_task.bpmn (size: 2526 bytes). Deployed process model bpmn/one_task.bpmn successful with key 2251799813685431. Deployed given process model , under key 2251799813685431! Handle zbchaos job [key: 2251799813685803] Running command with args: [verify instance-creation --partitionId 1] Connecting to zell-chaos Running experiment in self-managed environment. Successfully created port forwarding tunnel Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 4503599627374207 on partition 2, required partition 1. Send create process instance command, with BPMN process ID 'benchmark' and version '-1' (-1 means latest) [variables: '', awaitResult: false] Created process instance with key 2251799813689056 on partition 1, required partition 1. The steady-state was successfully verified! Instance 2251799813685255 [definition 2251799813685253 ] completed --- PASS: Test_ShouldBeAbleToRunExperiments (11.98s) PASS ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
✅ Depends on #268
related to #237
Migrates the follower restart experiment and tested with integration test
Log output: